Penalized Regression Approach Research Articles

Flexible variable selection in the presence of missing data.

In many applications, it is of interest to identify a parsimonious set of features, or panel, from multiple candidates that achieves a desired level of performance in predicting a response. This task is often complicated in practice by missing data arising from the sampling design or other random mechanisms. Most recent work on variable selection in missing data contexts relies in some part on a finite-dimensional statistical model, e.g., a generalized or penalized linear model. In cases where this model is misspecified, the selected variables may not all be truly scientifically relevant and can result in panels with suboptimal classification performance. To address this limitation, we propose a nonparametric variable selection algorithm combined with multiple imputation to develop flexible panels in the presence of missing-at-random data. We outline strategies based on the proposed algorithm that achieve control of commonly used error rates. Through simulations, we show that our proposal has good operating characteristics and results in panels with higher classification and variable selection performance compared to several existing penalized regression approaches in cases where a generalized linear model is misspecified. Finally, we use the proposed method to develop biomarker panels for separating pancreatic cysts with differing malignancy potential in a setting where complicated missingness in the biomarkers arose due to limited specimen volumes.

Using machine-learning to decode postoperative hip mortality Trends: Actionable insights from an extensive clinical dataset

BackgroundHip fractures are one of the most common injuries experienced by the general population. Despite advances in surgical techniques, postoperative mortality rates remain high. identifying relevant clinical factors associated with mortality is essential to preoperative risk stratification and tailored post-surgical interventions to improve patient outcomes. The purpose of this study aimed to identify preoperative risk factors and develop predictive models for increased hip fracture-related mortality within 30 days post-surgery, using one of the largest patient cohorts to date. MethodsData from the American College of Surgeons National Surgical Quality Improvement Program database, comprising 107,660 hip fracture patients treated with surgical fixation was used. A penalized regression approach, least absolute shrinkage and selection operator was employed to develop two predictive models: one using preoperative factors and the second incorporating both preoperative and postoperative factors. ResultsThe analysis identified 68 preoperative factor outcomes associated with 30-day mortality. The combined model revealed 84 relevant factors, showing strong predictive power for determining postoperative mortality, with an AUC of 0.83. ConclusionsThe study's comprehensive methodology provides risk assessment tools for clinicians to identify high-risk patients and optimize patient-specific care.

Dual statistical models link baseline visual attention measure to risk for significant symptomatic concussion in sports

Aim: Athletic pre-season testing can establish functional baseline for comparison following concussion. Whether impacts of future concussions may be foretold by such testing is little known. Materials & methods: Two sets of models for a significant burden of concussion were generated: a traditional approach using a series of logistic regressions, and a penalized regression approach using elastic net. Results: 3091 youth and adult athletes were baseline-assessed. 90 subsequently experienced concussion and 35 were still experiencing a significant burden of concussion when tested within two weeks. Both models associated prior history of head injury and visual attention-related metrics with a significant burden of concussion. Conclusion: Pre-season testing of visual attention may identify athletes who are at risk for significant sports-related concussion.

Clustered Coefficient Regression Models for Poisson Process with an Application to Seasonal Warranty Claim Data

Motivated by a product warranty claims dataset, we propose clustered coefficient regression models in a nonhomogeneous Poisson process for recurrent event data. The proposed method, referred as CLUPP, can estimate the group structure and parameters simultaneously. In our proposed method, a penalized regression approach is used to identify the group structure. Numerical studies show that the proposed approach can identify the group structure well, and outperforms traditional methods such as hierarchical clustering and K-means. We also establish theoretical properties, which show that the proposed estimators can converge to true parameters in high probability. In the end, we apply our proposed methods to the product warranty claims dataset, which achieve better prediction than the state-of-the-art methods.

Social Frailty Index: Development and validation of an index of social attributes predictive of mortality in older adults

While social characteristics are well-known predictors of mortality, prediction models rely almost exclusively on demographics, medical comorbidities, and function. Lacking an efficient way to summarize the prognostic impact of social factor, many studies exclude social factors altogether. Our objective was to develop and validate a summary measure of social risk and determine its ability to risk-stratify beyond traditional risk models.We examined participants in the Health and Retirement Study, a longitudinal, survey of US older adults. We developed the model from a comprehensive inventory of 183 social characteristics using least absolute shrinkage and selection operator, a penalized regression approach. Then, we assessed the predictive capacity of the model and its ability to improve on traditional prediction models.We studied 8,250 adults aged≥65 y. Within 4 y of the baseline interview, 22% had died. Drawn from 183 possible predictors, the Social Frailty Index included age, gender, and eight social predictors: neighborhood cleanliness, perceived control over financial situation, meeting with children less than yearly, not working for pay, active with children, volunteering, feeling isolated, and being treated with less courtesy or respect. In the validation cohort, predicted and observed mortality were strongly correlated. Additionally, the Social Frailty Index meaningfully risk-stratified participants beyond the Charlson score (medical comorbidity index) and the Lee Index (comorbidity and function model).The Social Frailty Index includes age, gender, and eight social characteristics and accurately risk-stratifies older adults. The model improves upon commonly used risk prediction tools and has application in clinical, population health, and research settings.

Machine learning models for predicting steroid-resistant of nephrotic syndrome.

In the absence of effective measures to predict steroid responsiveness, patients with nonhereditary steroid-resistant nephrotic syndrome (SRNS) have a significantly increased risk of progression to end-stage renal disease. In view of the poor outcomes of SRNS, it is urgent to identify the steroid responsiveness of idiopathic nephrotic syndrome (INS) early. To build a prediction model for SRNS, we collected 91 subjects; 57 of them had steroid-sensitive nephrotic syndrome, and the others had SRNS. For each subject, 87 clinical variables were measured. In general, only a small part of these variables is informative to SRNS. Thus, we proposed a new variable selection framework including a penalized regression approach (named MLR+TLP) to select variables having a linear effect on the SRNS and a nonparametric screening method (MAC) to select variables having a nonlinear marginal (joint) effect on the SRNS. Thereafter, considering the correlation between selected clinical variables, we used a stepwise method to build our final model for predicting SRNS. In addition, a statistical testing procedure is proposed to test the overfitting of the proposed model. Twenty-six clinical variables were selected to be informative to SRNS, and an SVM model was built to predict SRNS with a leave-one-out cross-validation (LOO-CV) accuracy of 95.2% (overfitting p value<0.005). To make the model more useful, we incorporate prior medical information into the model and consider the correlation between selected variables. Then, a reduced SVM model including only eight clinical variables (erythrocyte sedimentation rate, urine occult blood, percentage of neutrophils, immunoglobulin A, cholesterol, vinculin autoantibody, aspartate aminotransferase, and prolonged prothrombin time) was built to have a LOO-CV accuracy of 92.8% (overfitting p value<0.005). The validation cohort showed that the reduced model obtained an accuracy of 94.0% (overfitting p value<0.005), with a sensitivity of 90.0% and a specificity of 96.7%. Notably, vinculin autoantibody is the only podocyte autoantibody included in this model. It is linearly related to steroid responsiveness. Finally, our model is freely available as a user-friendly web tool at https://datalinkx.shinyapps.io/srns/. The SRNS prediction model constructed in this study comprehensively and objectively evaluates the internal conditions and disease status of INS patients and will provide scientific guidance for selecting treatment methods for children with nonhereditary SRNS.

Machine learning procedures for predictor variable selection for schoolwork-related anxiety: evidence from PISA 2015 mathematics, reading, and science assessments

BackgroundLarge-scale international studies offer researchers a rich source of data to examine the relationship among variables. Machine learning embodies a range of flexible statistical procedures to identify key indicators of a response variable among a collection of hundreds or even thousands of potential predictor variables. Among these, penalized regression approaches, including least absolute selection and shrinkage operator (LASSO) and elastic net (Enet), have been advanced as useful tools capable of handling large number of predictors for variable selection for model generation. While the utility of penalized regression within educational research is emerging, less application of these machine learning methods, including random forest, to predictor variable selection in large-scale international data appears in the literature. In response, this study compared LASSO, Enet, and random forest for predictor variable selection, including the traditional forward stepwise (FS) regression approach, for students’ test anxiety or, more specifically, schoolwork-related anxiety based on PISA 2015 data.MethodsPrediction of the three machine learning methods were compared for variable selection of 188 indicators of schoolwork-related anxiety. Data were based on US students (N = 5593) who participated in PISA 2015. With the exception of FS, LASSO, Enet, and random forest were iterated 100 times to consider the bias resulting from data-splitting to determine the selection or non-selection of each predictor. This resulted in the reporting of number of selected variables into the following five count categories: 1 or more, 25 or more, 50 or more, 75 or more, and all 100 iterations.ResultsLASSO and Enet both outperformed random forest but did not differ from one another in terms of prediction performance in 100 iterations of modeling. Correspondingly, LASSO was compared to FS in which, of the 188 predictors, 27 were identified as key indicators of schoolwork-related anxiety across 100 iterations, and 26 variables were also statistically significant with FS regression. Aligned with previous research, key indicators included personal, situational, and mathematics and reading achievement. Further, LASSO identified 28 variables (14.89%) statistically unrelated to schoolwork-related anxiety, which included indicators aligned to students’ academic- and non-academic behaviors.ConclusionsLASSO and Enet outperformed random forest and yielded comparable results in which determinants of schoolwork-related anxiety included personal and environmental factors, including achievement goals, sense of belonging, and confidence to explain scientific phenomenon. LASSO and FS also identified similar predictor variables related, as well as unrelated, to schoolwork-related anxiety. Aligned with previous research, females reported higher schoolwork-related anxiety than males. Mathematics achievement was negatively related to anxiety, whereas reading performance was positively associated with anxiety. This study also bears significance as one of the first penalized regression studies to incorporate sampling weights and reflect the complex sampling schemes of large-scale educational assessment data.

Investigation of Genetic Variants Associated with Tryptophan Metabolite Levels via Serotonin and Kynurenine Pathways in Patients with Bipolar Disorder.

The kynurenine pathway (KP) may play a role in the pathophysiology of bipolar disorder (BD). We conducted a genome-wide association study (GWAS) to identify genetic variants associated with the plasma levels of the metabolites of tryptophan (TRP) via the serotonin (5-HT) and kynurenine (KYN) pathways in 44 patients with BD and 45 healthy controls. We assessed whether variants that were differentially associated with metabolite levels based on the diagnostic status improved the prediction accuracy of BD using penalized regression approaches. We identified several genetic variants that were significantly associated with metabolites (5-HT, 5-hydroxytryptophan (5-HTP), TRP, and quinolinic acid (QA) or metabolite ratios (5-HTP/TRP and KYN/TRP) and for which the diagnostic status exerted a significant effect. The inclusion of genetic variants led to increased accuracy in the prediction of the BD diagnostic status. Specifically, we obtained an accuracy of 0.77 using Least Absolute Shrinkage and Selection Operator (LASSO) regression. The predictors retained as informative in this model included body mass index (BMI), the levels of TRP, QA, and 5-HT, the 5-HTP/TRP ratio, and genetic variants associated with the levels of QA (rs6827515, rs715692, rs425094, rs4645874, and rs77048355) and TRP (rs292212) or the 5-HTP/TRP ratio (rs7902231). In conclusion, our study identified statistically significant associations between metabolites of TRP via the 5-HT and KYN pathways and genetic variants at the genome-wide level. The discriminative performance of penalized regression models incorporating clinical, genetic, and metabolic predictors warrants a follow-up analysis of this panel of determinants.

What determines bitcoin liquidity? A penalized regression approach

ABSTRACT We investigate which factors contribute most to the liquidity of Bitcoin, using a diverse universe of candidate factors reflecting key developments in the crypto market and the global economy. The empirical analysis relies on three regularized linear regression methods, viz. LASSO, adaptive LASSO, and elastic net. We also apply a cross-fit partialing-out LASSO instrumental-variables regression model, as a supplementary approach to handle endogeneity. Findings reveal that trading volume and realized volatility of Bitcoin, cryptocurrency hacks, Ethereum liquidity, and public attention are the most common drivers of liquidity, irrespective of the penalized regression approach and liquidity proxy adopted. Our evidence confirms the paramountcy of cryptocurrency-specific factors over global economic and financial ones in influencing Bitcoin liquidity.

Handling high-dimensional data with missing values by modern machine learning techniques

High-dimensional data have been regarded as one of the most important types of big data in practice. It happens frequently in practice including genetic study, financial study, and geographical study. Missing data in high dimensional data analysis should be handled properly to reduce nonresponse bias. We discuss some modern machine learning techniques including penalized regression approaches, tree-based approaches, and deep learning (DL) for handling missing data with high dimensionality. Specifically, our proposed methods can be used for estimating general parameters of interest including population means and percentiles with imputation-based estimators, propensity score estimators, and doubly robust estimators. We compare those methods through some limited simulation studies and a real application. Both simulation studies and real application show the benefits of DL and XGboost approaches compared with other methods in terms of balancing bias and variance.

Comparing the Performance of Prediction Model of Ridge and Elastic Net in Correlated Dataset

Multicollinearity refers to a condition where high correlation between independent variables in linear regression model occurs.  In this case, using ordinary least squares (OLS) leads to unstable model. Some penalized regression approaches such as ridge and elastic-net regression can be applied to overcome the problem. Penalized regression estimates model by adding a constrain on the size of parameter regression. In this study, simulation dataset is generated, comprised of 100 observation and 95 independent variables with high correlation. This empirical study shows that elastic-net method outperforms the ridge regression and OLS.  In correlated dataset, the OLS is failed to produce a prediction model based on mean squared error (MSE)

Bayesian regularization for a nonstationary Gaussian linear mixed effects model.

In omics experiments, estimation and variable selection can involve thousands of proteins/genes observed from a relatively small number of subjects. Many regression regularization procedures have been developed for estimation and variable selection in such high-dimensional problems. However, approaches have predominantly focused on linear regression models that ignore correlation arising from long sequences of repeated measurements on the outcome. Our work is motivated by the need to identify proteomic biomarkers that improve the prediction of rapid lung-function decline for individuals with cystic fibrosis (CF) lung disease. We extend four Bayesian penalized regression approaches for a Gaussian linear mixed effects model with nonstationary covariance structure to account for the complicated structure of longitudinal lung function data while simultaneously estimating unknown parameters and selecting important protein isoforms to improve predictive performance. Different types of shrinkage priors are evaluated to induce variable selection in a fully Bayesian framework. The approaches are studied with simulations. We apply the proposed method to real proteomics and lung-function outcome data from our motivating CF study, identifying a set of relevant clinical/demographic predictors and a proteomic biomarker for rapid decline of lung function. We also illustrate the methods on CD4 yeast cell-cycle genomic data, confirming that the proposed method identifies transcription factors that have been highlighted in the literature for their importance as cell cycle transcription factors.

Robust priors for regularized regression

Induction benefits from useful priors. Penalized regression approaches, like ridge regression, shrink weights toward zero but zero association is usually not a sensible prior. Inspired by simple and robust decision heuristics humans use, we constructed non-zero priors for penalized regression models that provide robust and interpretable solutions across several tasks. Our approach enables estimates from a constrained model to serve as a prior for a more general model, yielding a principled way to interpolate between models of differing complexity. We successfully applied this approach to a number of decision and classification problems, as well as analyzing simulated brain imaging data. Models with robust priors had excellent worst-case performance. Solutions followed from the form of the heuristic that was used to derive the prior. These new algorithms can serve applications in data analysis and machine learning, as well as help in understanding how people transition from novice to expert performance.

Randomized boosting with multivariable base-learners for high-dimensional variable selection and prediction

BackgroundStatistical boosting is a computational approach to select and estimate interpretable prediction models for high-dimensional biomedical data, leading to implicit regularization and variable selection when combined with early stopping. Traditionally, the set of base-learners is fixed for all iterations and consists of simple regression learners including only one predictor variable at a time. Furthermore, the number of iterations is typically tuned by optimizing the predictive performance, leading to models which often include unnecessarily large numbers of noise variables.ResultsWe propose three consecutive extensions of classical component-wise gradient boosting. In the first extension, called Subspace Boosting (SubBoost), base-learners can consist of several variables, allowing for multivariable updates in a single iteration. To compensate for the larger flexibility, the ultimate selection of base-learners is based on information criteria leading to an automatic stopping of the algorithm. As the second extension, Random Subspace Boosting (RSubBoost) additionally includes a random preselection of base-learners in each iteration, enabling the scalability to high-dimensional data. In a third extension, called Adaptive Subspace Boosting (AdaSubBoost), an adaptive random preselection of base-learners is considered, focusing on base-learners which have proven to be predictive in previous iterations. Simulation results show that the multivariable updates in the three subspace algorithms are particularly beneficial in cases of high correlations among signal covariates. In several biomedical applications the proposed algorithms tend to yield sparser models than classical statistical boosting, while showing a very competitive predictive performance also compared to penalized regression approaches like the (relaxed) lasso and the elastic net.ConclusionsThe proposed randomized boosting approaches with multivariable base-learners are promising extensions of statistical boosting, particularly suited for highly-correlated and sparse high-dimensional settings. The incorporated selection of base-learners via information criteria induces automatic stopping of the algorithms, promoting sparser and more interpretable prediction models.

Jewel: A Novel Method for Joint Estimation of Gaussian Graphical Models

In this paper, we consider the problem of estimating multiple Gaussian Graphical Models from high-dimensional datasets. We assume that these datasets are sampled from different distributions with the same conditional independence structure, but not the same precision matrix. We propose jewel, a joint data estimation method that uses a node-wise penalized regression approach. In particular, jewel uses a group Lasso penalty to simultaneously guarantee the resulting adjacency matrix’s symmetry and the graphs’ joint learning. We solve the minimization problem using the group descend algorithm and propose two procedures for estimating the regularization parameter. Furthermore, we establish the estimator’s consistency property. Finally, we illustrate our estimator’s performance through simulated and real data examples on gene regulatory networks.

Derivation and validation of a prediction model for histopathologic fibrotic hypersensitivity pneumonitis

BackgroundClinical differentiation of fibrotic hypersensitivity pneumonitis (f-HP) remains challenging given variable and overlapping presentations with other fibrotic interstitial lung disease (f-ILD). ObjectiveWe derived a multivariable model for predicting histopathologic f-HP to better inform multidisciplinary team discussion (MDD) diagnosis, particularly when biopsy may be unsafe or cannot be achieved. MethodsPatients with histopathologically-defined f-HP and other overlapping f-ILD were reviewed for distinguishing clinical and radiological variables. Using elastic net logistic regression, a penalized regression approach to minimize overfitting, a clinical model built on non-invasive assessments was derived for the prediction of histopathologic f-HP. This model was then validated in an independently derived external cohort from three sites. ResultsThe derivation and validation cohorts consisted of 248 (84 cHP and 164 other f-ILD) and 157 (82 f-HP and 75 other f-ILD) histopathologically-defined patients, respectively (total study N = 405). Variables retained from the elastic net model included age in years (regression coefficient 0.033), male sex (−1.109), positive exposure history (1.318), percent predicted forced vital capacity (−0.021), radiologic peribronchovascular axial ILD distribution (0.199), mid (−0.22) or lower lobe (−0.839) craniocaudal or patchy (0.287) ILD distribution, upper (1.188) or equivalent upper and lower lobe (0.237) traction bronchiectasis, mosaic attenuation (1.164), and centrilobular nodules (2.045). Bias corrected AUC was 0.84 (standard error = 0.02) for the derivation cohort and 0.80 (CI 0.73–0.87) for the validation cohort. ConclusionsThis multivariable model demonstrated good predictive performance for delineating histopathologically-defined f-HP from other f-ILD as a means of avoiding or justifying biopsy and supporting MDD diagnostic confidence.

Risk spillover network structure learning for correlated financial assets: A directed acyclic graph approach

Using cross-asset return data in global financial markets, we propose a novel empirical framework to identify the causal structure of the asset risk spillover network. The joint return distribution of the global financial system can be characterized using a directed acyclic graph approach. However, since assets tend to be highly correlated during market turbulence, when adopting a nodewise penalized regression approach for neighborhood estimation, parameter estimates will receive large standard errors, and edges cannot be reliably estimated. In this work, we propose a two-stage approach for directed acyclic graph skeleton estimation for highly correlated variables. In the first stage, a variable screening ensemble is incorporated into the sparse partial least squares regression method to both reduce the size of the active variables set and impose an adaptive penalization on the weight vectors. In the second stage, a modified PC algorithm based on Gram-Schmidt orthogonalization is applied to remove the false positive edges. Simulation studies are conducted to demonstrate the effectiveness of the proposed method. Finally, we apply our method to analyze the asset risk spillover channels for international financial assets during the COVID-19 pandemic.

Group structure detection for a high‐dimensional panel data model

This article investigates latent group structures in a high‐dimensional panel data model. It is assumed that subjects are classified into unobserved groups, where the fixed effects for each subject are the same within a group but heterogeneous across different groups. A penalized regression approach is proposed to identify latent group structures and significant covariates simultaneously. A new algorithm is proposed to optimize the objective function. When the sample size goes to infinity, it is shown that the proposed estimator recovers latent group structures and important covariates with probability approaching one. The finite sample performance of the proposed estimator is evaluated through Monte Carlo studies and illustrated using a real dataset.

Methylation-eQTL analysis in cancer research.

DNA methylation is a key epigenetic factor regulating gene expression. While promoter methylation has been well studied, recent publications have revealed that functionally important methylation also occurs in intergenic and distal regions, and varies across genes and tissue types. Given the growing importance of inter-platform integrative genomic analyses, there is an urgent need to develop methods to discover and characterize gene-level relationships between methylation and expression. We introduce a novel sequential penalized regression approach to identify methylation-expression quantitative trait loci (methyl-eQTLs), a term that we have coined to represent, for each gene and tissue type, a sparse set of CpG loci best explaining gene expression and accompanying weights indicating direction and strength of association. Using TCGA and MD Anderson colorectal cohorts to build and validate our models, we demonstrate our strategy better explains expression variability than current commonly used gene-level methylation summaries. The methyl-eQTLs identified by our approach can be used to construct gene-level methylation summaries that are maximally correlated with gene expression for use in integrative models, and produce a tissue-specific summary of which genes appear to be strongly regulated by methylation. Our results introduce an important resource to the biomedical community for integrative genomics analyses involving DNA methylation. We produce an R Shiny app (https://rstudio-prd-c1.pmacs.upenn.edu/methyl-eQTL/) that interactively presents methyl-eQTL results for colorectal, breast and pancreatic cancer. The source R code for this work is provided in the Supplementary Material. Supplementary data are available at Bioinformatics online.

Functional principal component analysis for global sensitivity analysis of model with spatial output

Motivated by risk assessment of coastal flooding, we consider time-consuming simulators with a spatial output. The aim is to perform sensitivity analysis (SA), quantifying the influence of input parameters on the output. There are three main issues. First, due to computational time, standard SA techniques cannot be directly applied on the simulator. Second, the output is infinite dimensional, or at least high dimensional if the output is discretized. Third, the spatial output is non-stationary and exhibits strong local variations.We show that all these issues can be addressed all together by using functional PCA (FPCA). We first specify a functional basis, such as wavelets or B-splines, designed to handle local variations. Secondly, we select the most influential basis terms, either with an energy criterion after basis orthonormalization, or directly on the original basis with a penalized regression approach. Then FPCA further reduces dimension by doing PCA on the most influential basis coefficients, with an ad-hoc metric. Finally, fast-to-evaluate metamodels are built on the few selected principal components. They provide a proxy on which SA can be done. As a by-product, we obtain analytical formulas for variance-based sensitivity indices, generalizing known formula assuming orthonormality of basis functions.